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(57) Abstract 

A method for dynamic video annotation among a plurality of users utilizes a programmable computer and comprises die steps of: 
selecting a network service coupled to the computer; performing one of (a) starting and (b) joining a collaborative session among the users; 
loading one of (a) a video and (b) a recorded annotation file; performing annotation of at least one of graphical, text, and audio annotation; 

and storing said collaborative session. 
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APPARATUS AND METHOD FOR 
COLLABORATIVE DYNAMIC VIDEO ANNOTATION 

5 FIELD OF TOE INVENTION 

The present invention relates to the field of collaborative video annotation, and 
more specifically, to apparatus for enabling muitiple users to share their views about a 
video content 

10 BACKGROUND QFTOEINVENnON 

Reference is niade to a patent application entided Method and Apparatus fin- 
Creating Dynamic Object Markers in a Video Clip being filed on even date hoewith, 
and assigned to the same assignee as the present application, and whereof the disclosure 
is herein incorporated by refermce to the extent it is not incompatible with the present 

15 applicatioa 

A situation can arise wherein two or more users wish to communicate in 
refeenceto a common deject, fi^rexantple, in reference to a video. An example of this 
could be where a soccer team coach wishes to consult with a colleague to seek advice. 
The soccer team ooadi mi^ wish to show a taped video of a game and ask the 

20 colleague to explain, using the video, why one team &iled to score in a given attack 
situatioa In additk>n, the coach might wish to record this discussion and show it later 
to other coaches to get more ofmuons. 

In anmfaer scenario, a student could be taking a training course being given at a 
remote bcadon from where a course in^ctor is k)cated. It may be that the student 

25 cannot understand a procedure being taught in the course. The student can then call die 
instructor over the Internet phone to find out how such a procedure shouM be 
perfimned. The instructor can first browse through the training video together with the 
student to find the dip where the difficulty can be identified. The student may then adc 
various questbns of the instnictor about that procedure. For exanq>Ie, the mstnictor 

30 may then decide to show the student another video, which offers more detailed 
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infonnation. The instructor may then annotate this video using collaborative video 
annotation tools to explain to the student how this procedure should be performed. 

A need exists for systems and products to provide services such as those 
described above. One such product is the Sprint's Drums system which allows two 
5 users to view video simultaneously by using the Shared Movie Player that runs on 
Silicon Graphics, Inc. computers. The shared video playback stans with one of the 
users sending the video file in SGI Movie Player format to be shared with the other 
user. Once the complete video has been transferred, either of the two users can 
initiate video playback. The playback control is also shared. Either of the two users 
10 can pause the video, jump to a random position in the video by use of a scrollbar, 
or playback video in reverse direction. 

However, the Shared Movie Player generally does not provide certain 
features such as graphical annotation on top of the video window. In order to add 
graphical annotations, the user will have to pause the video, copy and paste the 
15 current frame to an ordinary shared whiteboard application. 

A Tele^Pointer, which is a device for controlling the appearance and 
position of a cursor or points on computer displays from a remote location, is 
also typically not provided; the video window itself is not shared and the users do 
not have any means for sharing a pointing device in either play or pause mode. 
20 Generally an integrated audio conferencing mixing confmnce audio with 

video sound-track is not provided: a regular telephone connection is typically used 
for us^-to-user dialogues. 

In such system, recording/playback of shared playback session is typically 
not provided nor is Multi-user conferencing: the Shared Movie Player only works 
25 for point-to-ppint conferencing. 

Yet another product. Creative Partner from eModon[2,3], contains three video 
annotation tools, but annotation has to be recorded off-line; there being no on-line 
collaboratbn support and during a period when the video playback is paused. These 
three annotattcm tools inchide one for graphical annotation^ one for text annotation, and 
30 one for audio annotatioa The Creative Partha* video player allows the user to control 
video piaybadc and to invdce one of the three annotation tools provided. Annotation 
can only be attached to the video in pause mode. The user selects the appropriate 
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annotation tool, points to a given image coordinate on the video frame to which the 
annotation is attached. The user will thai be able to record the given annotation. The 
annotation is not related to a video segment, it being only related to the frame to whidi 
it was attached. During playback, the Creative Partno' video player will pause at any 
5 video frame having annotations attached. The user will have to activate playbadc of 
audio annotations and to resume video playback. The annotations are removed from the 
video window, once video playback is resumed. 

Helpful background information can be found in U.S. Patent No. 5,600,775, 
issued February 4, 1997 in die names of King et al. and ENTITLED METHOD AND 
10 APPARATUS FOR ANNOTATING FULL MOTION VIDEO AND OTHER 
INDEXED DATA STRUCTURES, and at Internet site 
http://www.emotion.com/htnil/creativq3artner product page.html. 

SUMMARY OF THE INVENTION 
15 It is hmin recognized that there is a continuing need for an sqpparatus to 

provide the foltowing fimctional features which relate to aspects of the imsent 
inventimt 

On-line nxulti*point group discussions on video content over heterograeous 
networks with Tde-Pointer support; 
20 synchronized video playback, overl^d with voice comments as wdl as 

dynamic graphical annotation during group discussion: 

dynamic adjustment of playback speed during synchronized video playback 
and recordii^ of group discussion sessions; 

in response to an existing recorded annotation session, conduct subsequent 
25 on-line multi-point group discussions and annotate during synchronized playback 
of recorded annotations; and 

attadmiem of any tool to help browsing of video content and creation of 
dynamic markers for static as well as dynamic objects. 

Jn coOabofative dynamic video annotation applications, it is generally 
30 considered unlikdy that the participants in the group discussion will either own the 
same type of computer equipment or be physically be present in the same building. It is 
also considered in general unlikely that each participant can be required to have a 
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cormectton of equal data-rate to the Internet. 

It is herein recognized that a desrabie solution to the above problems should 
provide enough flexibility to overcome problems due to the existence of heterogeneous 
environments. For example; a desirable solution, in accordance with an aspect of the 
5 present invention, can allow people to use a Public Switched Telecommunications 
Network (PSTN), if quality of audio is of concern, and the Internet Phone (IP), if cost is 
a greato- concern. 

Tde-Pointers are an important part of a group discussion. Without them, it is 
practically impossible to know what each participant is pointing at. It is heran 

10 recognized that since video is a dynamic document, it is more helpful, in acccmianoe 
with an aspect of the present invention, if each participant can make their own cursor 
visible on the screens of other participants. 

Since video content is dtfBcult to describe verbally, it is herein recognized, in 
accordance with an aspect of the present invention, that it is important for all 

15 partidpants to see the same video frame at die same time. It is also required that ai^ 
participant be able to annotate, such as with graphical drawings or texts^ on top of a 
video frame independently of the state of the video player and for all participants to see 
the same annotations on each of their screen at the same time. In addition, since 
partidpants are usually located in respective remote locations, the ability to have fiill 

20 duplex multi-point voice communication is considered essential. The system should 
preferably also be able to mix the audio track in the video with the voice of all 
participants all die time. 

Sometimes^ annotating dmultaneously with voice and gn4>hical drawings while 
the video is being pl^ed is not a very straightforward task. It is herein recognized that» 

25 m accordance widi an aspect ofdie present invention, any participant shoukl be able to 
(fynamically adjust the shared video playback speed durmg the group discussioa The 
entire group discussion should preferably be able to be recorded and played back in the 
same sequence in which it happened. The recording should preferably indude all VCR 
commands^ gnqphical drawings/texts, and voice comments, time-stamped for later 

30 playback. 

Playback of a recorded annotation can occur in a stand-alone mode or an on- 
line collaboration mode. In the second case, the playback of the recorded annotation 
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should be synchronized among all participants. In accordance with an aspect of the 
present invention, any participant should be able to annotate while the playback is 
going on and record new annotations in a s^arate record. 

In accordance with an aspect of the present invention, during a collaborative 
5 dynamic video annotation, any participant is able to use add-on tools to fecUttate the 
discussion. One such tool is a video browser which allows a user to jump to random 
points in the video. While the video is being played, proper places in the tool are 
preferably be highlighted to reflect the frame that is currently being played on the 
screen of each participant Another tool that may be implemented is the tool that allows 

10 any participant to create dynamic object markers. A dynamic object marker is a 
graphical drawing (usually assanbled from polygons) that highlights the location of an 
object of interest in a video fiiame. The dynamic marker indicates dynamic objects and 
dynamic parameters relating to a steady object. Since the location of a dynamic object 
generally diai^es from frame to frame, this tool is provided in accordance with die 

15 invention to help locate this object in all frames between a selected pair of video 
fiames.Thistod, when invoked, wall create a marker for all frames b^ween a selected 
pair of video frames. The system preferably also provides a tool that creates (fynamic 
maricers finr steady objects. This is usefril when the object of interest contains dynamic 
infiinnation parameters; be example; current flow in an electrically conductive wire. 

20 In aooordanoe widi the present invention, a computer based system or apparatus 

provides collaborative dynamic video annotation, recc^rding of such a collaborative 
session, synchronized playback of such a recorded annotation, and annotation/recording 
duriiig playback of a recorded annotation. The apparatus comprises a computer 
readable storage ii»dhmi having a computer program stored thereon performing the 

25 steps of: (a) choosir^ a network service and starting or joining a conference; 0>) loading 
a video or a recorded annotation file; (c) performing simultaneous graphical, text, and 
audio annotation; widi the support of tdl-pointers, VCR controls, video browses^ and 
dynamic marter creation tods; and (d) recording of a collaborative session. 

The system in accordance with the invention is a client/server modd. The dient 

30 essentially oonqprises a diared multi-media player with synchronized multi-point VCR 
control. The window that displays live video frames is also a drawing board where 
(fifferent people can place graphical objects from different dients at the same time. 
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In addition, the system provides multi-point fUll duplex voice connection and the joint 
voice comments are mixed vAth the audio track in the video currently being played. 

The n^work sesmce in St^ (a) includes any of the Internet TCP/IP, IPX, 
Modem, and Serial connection. The video file described in Step (b) is located locally 
5 with the client Step (c) can be ^ecuted independently of whedier the video player is in 
the play or pause modes and the playback among multiple clients is loosely 
synchronized, as explamed below, v^ile the speed of the video piaybadc can be 
adjusted dynamically during synchronized playback. Step (d) recoitis all visible 
activities in the collaborative session, such as lines, texts, Tde-Pointers, markers, as 

1 0 wdl as voice comments exchanged during the session. 

It is important for any shared applications to serialize all events that occurred 
during a session. In order to implement a shared but synchronized multi-media player 
while giving all participants an equal access to the VCR control, tiie player action is 
ddayed while die VCR button is first pressed. Instead ofinterpreting a VCR command 

15 immediately on a dient machine, a VCR command is sent to the server, serialized 
sequentially, and sent back to all client machines. Only after having received meiged 
VCR commands fiom the server does the player then take action. 

In order to synchronize video playback among all participants, some 
cooperation is required b^ween all video players. This can be done on a frame^- 

20 fiame basis, which is very cosdy and difficuh to realize in reality without sacrificing the 
plqdmck quality. In accordance with the present invention^ it is considered preferable 
to synchronize on a VCR command-by-command basis. The current fi-ame at the time 
the VCR control button is first pressed is recorded and each traditicmal VCR command 
is converted into one of the two types of new VCR commands, namely "JUMP-PLAY 

25 fiame-number firame^e" and **JUMP-PAUSE frame-number". For example, siqipose 
a PLAY button is pressed while the player is on fiame 300. The message sent to the 
server and eventually received by all clients will be "JUMP-PLAY 300 LO*" instead of 
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"PLAY"*, that is, the VCR will seek to frame 300 and then start playing the video at 
nonnalqpeed. 

Another component of the present invention is the ability to mix audio ^gnals 
and to overlay graphics with video frames. Some known audio/graphic hardware, such 
5 as Parallax board offers such a functionality. Even if a graphic overlay is supported by 
hardware, the system still needs to handle the drawing of Tele-Pointers and gr^hical 
aiuiotations sq)arately. This is because Tele-Pointers may occhide grq)hical 
aruiotations and graphical armotations need to be restored once Tele-Pointers move 
away fifom thdr current position. . 

10 In order for the system to play a recorded annotation session synchronously on 

the screens of all partidpants, it is only necessary to record, with.timestamp, aU the 
messages that readied the server during the discussion session and siend them to all 
clients acconfing to the timestamp during playback. This will also allow each 
participant to annotate during playback of recorded annotation. 

15 In aooordanoe with an aspect of the invention, a method for dyiuunic video 

amKytation among a phoality of users at respective locations, utilizmg progranmiabie 
conqniter apparatus with information storage and retrieval capability, the method 
cornprises the steps of sdecting a n^ork sersdce coupled to the computer, pcfforming 
(Mie of (a) starting and (b) joining a collaborative session among the users; loadirig one 

20 of (a) a video and (b) a recorded annotation file; performing annotation of at least one 
of grapMcat text, and audio armotation; and storing the collaborative session. 

In accordance witii another Bspecx of the invention a method for dynamic video 
aimotaticMi amor^ a pbirality of users, utilizing progranunable computer ansaratus with 
information ston^ge and retrieval capability, the method comprises the steps o£ 

25 sdecting a network service coupled to die computer, joining a collaborative session 
amoiig the users; k)admg one of (a) a video and (b) a recorded annotatim fil^ 
pei formii g aimotalkm of at least one of graphical, text, and audio anitotatior^ and 
storing the collaborative sessioa 

In accordance with another aspect of the invention apparatus for dynamic video 

30 annotatbn amorig a phirality of users; the apparatus comprises prograrrmmble computer 
apparatus with information storage and retrieval capability; a user inter&ce coupled to 
the compaxer apparatus for performing selection of a network service, a user inteifice 
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coupled to the computer apparatus for performing one of (a) starting and (b) joining a 
collaborative session among the users; a user interface coupled to the computer 
apparatus for loading one of (a) a video and (b) a recorded annotation file; a user 
inter&ce coupled to the compute apparatus for performing annotation of at least one of 
5 graphical, text, and audio annotation; and a user inter&ce coupled to the computer 
apparatus for storing the collaborative session. 

In accordance with another aspect of the invention apparatus for enabling a 
plundity of users at respective locations to participate in a collaborative session 
r^ardif^ content of a video; to record sudi a collaborative session; to annotate/recofd 

10 during playback of a recorded session; and to play back synchronously such a recorded 
annotated session, wherein the apparatus comprises a shared video player^recorder 
function (VCR) available to each of the users, with multi-point VCR control exhibiting 
dynamic speed adjustment, and an ability to show dynamic markm; a function by 
which any of the users can play or stop the video; jump to a different location in the 

15 video; dynannicaUy change video play speed; the shared video player/recorder fiuictton 
available to each of the users being synchronized at the same video fiame whenever 
any VCR activity occurs; and apparatus for displaying a dynamic marker when a firame 
to vAoch such a marker is attached is displayed. 

In accordance with another aspect of the invention, apparatus for enabling a 

20 phirality of users at respective locations to partidpate in a collaborative session 
regardirig content of a video; to record such a collaborative session; to annotateAnecord 
durh^ playback of a recorded session; and to play back synchronously such a recmted 
armotated session, wheran the apparatus comprises a shared video playerAecorder 
fiuKtion (VCR) exfaibitmg a window available to each of the users, with muhiiXMnt 

25 VCR control exhibttii^ dynamic speed adjustment, and an ability to show dynamic 
markers; a fiinction by which any of the users can play or stop the video; jump to a 
diffeient k)catbn in the video; dynamically charge video play speed; the shared video 
pbyer/recorder fimction available to each of the users being synchronized at the same 
video fiame whenever any VCR acdvity occurs; apparatus for displaying a dynamic 

30 marker when a fiame to which such a marker is attached is displayed; and die shared 
video idsiyeri^ecorder fiinctk)n mndow acting as a shared ^iteboard widi Tde-Pdnter 
support for siqiportingfi^hand drawing and text 
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In accordance with another aspect of the invention a method for dynamic video 
annotation among a plurality of users utilizes a programmable computer and comprises 
the steps of: selecting a network service coupled to the computer, performing one of (a) 
starting and (b) joining a collaborative session among the users; loading one of (a) a 
5 video and (b) a reootded annotation file; performing annotation of at least one of 
graphical, text and audio annotation; and storing the collaborative session. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The invention will be better understood 6x>m the following detailed description 
10 in conjunction with the Drawing, in which 

Fig.1 is a diagram illustrating a collaborative dynamic annotation session over 
Public Switched Telecommunications Networic (PSTN) in accordance wiA the 
invention; 

Fig. 2 is a diagram illustrating a collaborative dynamic annotation session over 
15 Internet Phone (IP) in accordance with the invention; 

Fig. 3 is a diagram illustratii^ a front-end user inter&ce in accordance with the 
invention; 

Fig. 4 is a diaigram illustratii^ a main client user intergtce in accordance with 
the invention; 

20 Fig. 5 badiagram illustrating a user inter&ce in accordance with the invention 

for deleting dynamic markers by name; 

Fig, 6 is a diagram illustrating a video browser inter&ce in accordance with the 
invention; 

Fig. 7 b a diagram illustrating a dynamic object maricer creation tool inteifiice 
25 in accordance with the invendon; 

Fig. 8 is a diagram ilhistrating the system architecture in accordance with the 
invention; 

Rg. 9 is a diagram illustrating the system message flow in accordance with the 
invention; and 

30 fig. 10 is a diagram iUustradng an example that shows how event messages are 

communicated in accordance widi the invention. 
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DFTAn FD nRSCRIPTION OF THE PREFERRED EMBQDIMK^JTS 
The invention features a shared "video player" with multi-point VCR controi, 
dynansic speed adjustment, and the ability to show (^amic markers. It is emphasized 
however that the '"video player^ is contemplated in the present invention to be either (a) 
5 a simulated video player fiinction provided by a computer with stored video in its 
memory and simulated video player controls or (b) an actual VCR operated in 
conjunction with the rest of the system. Accordingly, it is herdn intended for 
simplidty and convenience to refer in the present patent application to a video player, 
video player/recorder, video recorder/player or VCR, by which is meant either a 

10 simulated or an actual VCSL or video player/recorder, or as a VCR function or a video 
recofder/player fiinction, according as a particular embodiment may be constituted. 

It is also understood that the qiparatus may utilize a television receiver 
qiparatus forthe video display fiinction, in association with a computer and a simulated 
or actual hardware to provide the functions herein disclosed. 

15 Fig.1 shows an embodiment illustrating how the invention is utilized in 

coiyunction with a public switched telephone network. A tdephone switch 100 is 
coupled to a computer 130, which in turn is coupled by way of a local area network 
(LAN) to each of a client set-up, 102 and 104. Gient set-up5l02 and 104 are eadi 
ecpiipped with a microphone, respectively. Telephone switch 1 00 is coupled to a 

20 cdlaborative dynamic video aniu»ation server 120 by way of acomputer modem 115. 
Tdephone switch 100 is also coupled a modem 1 1 5. Telephone switch 100 is coupled 
to telephones 110 and a monitor. Further computers, monitors, speakers, and 
tdephones amilar to those shown in Fig. I may be coupled to the ^em as shown, 
atdmgh not illustrated in Fig. 1. 

25 Tdefrixme switch 100 is responsible for handling a tdephone conference. In 

this sebupy participants in the conference can also use an DP-PSTN (Internet phone - 
Ptd>lic switched tdefrfione mtwoiic). bridge 130 to allow Intranet Phone users to use the 
puUic switched tdephone network. The collaborative dynamic video annotation server 
120, in accordance with the invention, handles all syston messages, such as tiiose 

30 marked by reference numeral 1030 in Fig. 10, sent by clients over the Interna Server 
120 also connects to the tdephone conference switch 100 by a computer modem 115. 
There are two types of clients or paitidpants. One utilizes a r^lar tdqihone 
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connection 1 10 and the other utilizes LAN coupled Intrant Phone connections. Both 
types of clients are equipped with a mouse 140, a k^oard 155, a pair of speakers 126, 
a monitor 160, and a main computer 108, coupled as shown in Fig. L For a computer 
equipped with an Intranet phone connection, a microphone 125 is required. 
5 An embodiment in Fig.2 shows the invention as utilized in an Internet Phone 

environment In this setup, the 200, providing the functionalities herein described and 
coupled to the Internet, handles all event messages sent by clients over the Internet. 
This server also acts like a digital phone switch, mixing all or part of voice comments 
spoken by confisrence participants and broadcasting the mixed signals back to all 
10 dients. All clients are each equipped with a mouse 220, a ke/board 215, a pair of 
speakers 230, a microphone 225, a monitor 205, and a main computer 210 coupled to 
diebitemet 

Fig.3 illustrates an embodim^ of a fifont-end interface in accordance wid) die 
invention, including optionally a telephone dialer button. In accordance with an 

15 embodiment of the invention, network service differences are hidden under a software 
layer through the use of Microsoft's DirectPtay or any software that implements die 
T.123 standard* 300 shows an interface that asks the user to select v^ch network 
service to use. The front-end saver side user inter&ce is 310 which pronqyts the 
usei/qperatorto sded a service provkier. The client side inter&ce is 320, whidi selects 

20 a service provkier and eventually die collaborative dynamic video annotation server. 

I4g.4 is shows an embodiment of a main client user interface in accordance 
widitheinventi(ML A user first sdects a video file or a recorded annotation file to load 
using interfice 400, wfafle inter&ce 410 gives brief summary instructions on its usage. 
Video fiames are displayed in a window 498. A button 408 (stq)), 412(iA^X 

25 416CpauseX 420(fist fbrwardX 424(&5t rewindX 428(step fbrwardX and 432 (step 
backwarcO provkle the basic VCR comrd functions for playing the video. A slider 436 
shows the cunent video position and also provkles a means for a user to randomly jump 
to any fi-ame m the video. Text display 440 displays the current frame number over die 
total fiame mimber comprised in the video. A user can dynamically adjust the {day 

30 speed by moving a slider 444 or by pressing the key + , normally provided on tte 
standard keyboard, for increasing current firame rate by 0. 1 fiame/secand thekey-for 
(tecreasing current firame rate by 0.1 fiame/sec. The current firame rate is displayed in 
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448. 

In order to allow a user to controi the playback of a recorded annotation, while 
pemutting them to diange the course of video playback a second set of VCR control, 
4S2(reoord), 4S6(stop), 460(playX 464(pause), is provided on the main client user 
5 inter&ce. These two sets of VCR controls are active simultaneously. However, v/hsn 
any of the first set of buttons (408, 412, 416, 420, 424, 428, 432) are pressed, the 
playback of the recorded annotation will automatically be stopped. Similarly, if a user 
starts playing a recorded annotation while the first set of VCR control is active^ the 
sy^em will also stop the current video playback activity. An reception is the recording 
10 activity, Initton 4S2, which will not be stopped even if the user presses any of the first 
set of buttons. 

In accordance with the invention, a user can make a fi-ee-hand drawing by 
holding down the left mouse button while moving the mouse cursor on the video fi'ame 
window 498. The lines will be shown on all clients' video fi^es in the same color as 

15 selected by the system or by the user on a per-user basis. A user can also type a text 
strii^ on the video fi:ame window 498 by first right clicking the mouse buttoa. When 
this hiqppens, the system will pop up a text edit window only on that particular client's 
screen. The user can then type any angle or multiple line text strings and didc on the 
OK hiitiHi when ready. Only at that time; the typed string(s) will appear on all clients' 

20 video fiames at the place where the right mouse click occurred Note that durit^ the 
typing process; the conference activities can continue. A user can also turn on thdr own 
Tele-Pointer to enable other participants to know v^ere they are pofaiting the mouse. 
This is accomplished by pressii^ die button 480. Inordernottooonfiiseauso-withtoo 
many graphical objects on screen, all graphical annotations, text annotations, and Tde- 

25 Pointers relating to the same user are drawn in the same color 

The system also aUows a user to erase the content of all graphical annotations 
by pressing button 472, the content of all text annotations by pres»ng button 468 in the 
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described embodiment, and the record of certain dynamic mark^ by pressing button 
476, 

Three attached tools are shown in the main client user inter&ce in Hg. 4. They 
are a video browsa* 486, a cfynamic marker creation tool for dynamic objects 490, a 
5 dynamic marker creation tool for steady (or static) objects 494. To start the first two 
tools, just press the conresponding buttons. To start the third tool, a us^ has to first 
pause the video, then click on button 494, and finally draw a free-hand curve on the 
video frame window 498. 

Fig. 5 illustrates an embodiment of a user inter&ce in accordance with the 

10 invention for deleting dynamic markers by name. As noted above, a dynamic maiker 
indicates the location of a dynamic object or the dynamic parameter peitaining to a 
staticobject EacA name is entered by the user who creates the marker. A list of maiker 
names (500) is showa A user can move a scroll bar (540) to view the whole list, select 
aname frcHn the list, and dick on a button 510 to delete it After ddetii^ all unwanted 

15 markers, the user can cUdc on a button 520 to terminate this pop-up window, ffthe user 
starts this pop-up window by mistake, they can click on a button 530 to tenninate lUs 
window. 

Rg. 6 iUustrates the video browser tool. This tool displays the first fiame of 
each shot in thumbnail picture 600. A user can use a scroll bar 630 to quickly glance 
20 through all thumbnail picbires and dick on button 6 1 0 to quit the video browser. While 
the video is friaying, the corresponding shot of the current video firnne will be 
synchronously highlighted in red firune 620. 

Fig. 7 is a diagram that illustrates the tool for creating dynamic object markers. 
This tool provides a separate video player 748, a slider 728, a current fiame indicator 
732, a video browser 768, and a cross-section viewer 752 to assist a user in finding a 
dip of interest After sdecting an IN point in the video by pressing button 736 and an 
OUT poitA in the video by presang button 740, the video frame cmresponding to die 
IN pcnm win be disfriayed in window 700 and the Oinr poim fia^ 
window 704. The cross-secdon inu^ 756 is generated directly from the video by (1) 
sampling the nncUIe row and the middle column from each every images (2) coUectirsg 
all samples over time, (3) combining them into one image, and (4) finally s^mentiiig 
the image into at least two bands according to the list of detected shots. This 
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rqmesentation provides a level of abstraction that reveals the continuity of video 
frames. A scroll bar 764 allows for a quick glance, while the current frame indicator 
760 is synchronized with the position of the video. 

Once the IN and the OUT point video frames are displayed on window 700 and 
5 704, a user can draw pbly-lines , bdng a sequence of lines such as shown by reference 
numeral 724 in Fig. 7, to oudine the boundary of the object in each window. If there is 
a mistake, the user can erase these lines by pressing button 712. After the poly-lines are 
drawn, a user can type a name in box 744 fr)r the marker and click on button 708 to ask 
the server to extraa the boundary of die same object in all frames between the IN and 

10 the OUT point The same steps can be repeated undl the button 716 is pressed. Theuser 
can also cancel operation of this tool by pressing button 720. 

Fig. 8 illustrates the system architecture for an embodiment in accordance with 
the principles of the invention.. The Collaborative Video Annotation Server 800 
receives messages from each session manager 812, serializes them, and re-sends tiiem 

15 to every session man^. It also manages the conference, keq)ing a record of all 
partidpantsas well asthe st^of the conftrafice. Such state information includes video 
pl^er state, inchiding play/pause and current fiBme number, Tde-Pomter state, 
including show/hide and current portion, annotations, including graphical and text 
currently on screen, video filename being loaded, and dynamic object markers (marker 

^ coordinates and associated fiamesw It is also re^onsible for bringing a new participant 
up to the current state of the conference and for recording a collaborative discussion 
. into a single script file for later playback. The final overlaid videograms are 
represented by reftrenoe numeral 852 and the recorded annotation by 804. 

The sessbn maaagBt 812 serves as the mediator between the Server 800, widi 

25 wluch it is coupled by way of the Internet, and the rest of client modules, 8 16, 820, 824, 
828, 83Z It acts like an intellig^ message router, transforming mouse^keyboard 
events firmi user interfece 820 into system messages and sending them to the 
collaborative video annotation server 800, forwarding system messages from a tt a dm i 
tools 832 to the collaborative video annotation server 800, and suppres^ng^distributing 

30 server messages to the rest of client modules. The system messages include all VCR- 
related commands, cursor positions, Tele-Pointer commands, annotation, both 
graphical and 
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text records, and annotation commands. The session manager 812 suppresses all local 

annotation record messages and all local Tele-Pointer commands. 

The Tele-Pointer 816 receives colleagues' active pointer positions as well as 

commands and draws them on the Tele-Pointer overlay plane 840. The Tde-Pointer 
5 conunands inchide Show and Hide. The system maintains a color scheme so that the 

pointer and the annotation fix)m the same participant will be drawn in the same color 

distinctive fix>m eadi other. 

The video player 828 decodes video into uncompressed audio and video frame 

and responds to a variety of VCR conmiands sent by the session manage 812. Due to 
10 the need for synchronized playback among all participants, it is preferred to map all 

traditional VCR-rdated commands into die following new commands: 

PLAY->JUMP.PLAYCIIRRENT.FRAN4E 1.0 

PAUSE «=> JUl^-PAUSE CURRENT-FRAME 

STEP FORWARD => JUMP-PAUSE CURRENT-FRAME+l 
15 STEP BACKWARD => JUMP-PAUSE CURRENT-FRAME- 1 

FAST FORWARD =>JUMP.PL AY CURRENT-FRAME 2.0 

FAST REWIND => JUMP-PLAY CURRENT-FRAME -2.0 

As will be noted, the system has created only two distinctive types of VCR 

commands, namely JUMP4>LAY frame-number frame*rate and JUMP-PAUSE fiame- 
20 number. To sappM the need of dynamic adjustment of playback speed, die system 

adds two new VCR-rdated fiincdons, + and and maps them to JUMP-PLAY by the 

following ways: 

+ => JUMP-PLAY CURRENT-FRAME CURRENT-RATE+0. 1 
- => JUMP-PLAY CURRENT-FRAME CURRENT-RATE-0. 1 

25 

User inter&ce 820 provides the elements of the player interfice such as VCR 
controls^ Tel&*Pointer controls^ the ability to launch attached tools 832» and monitors 
any mouse/k^board events. MouseAcqrboard events rdated to annotadon are sent to 
die video annotator 824, whereof the fonction is described bek>w« while all 
30 mouse/k^board events are sent to the session manager 812 for fiuther interpretatba 
As was mentioned diove; to draw any graphical annotation, the participant should hoM 
down the left buttm, ^^e moving the mouse. To draw a text annotation, the 
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partidpant should click on the right button and enter a text string on the popped-up box. 

The Audio Mixer 856 mixes the uncompressed audio signal generated by the 
video player with the audio output from the unified computer/Internet telephony 
inter&ce, and sends it to the speaker. 
5 The graphics overlay/Image mixer 848 overiays on top of the decoded video 

frame the graphical annotation, text annotation, and Tele-Pointer icons. This overlay 
has to be done e£Bciently and effectively because these have to be updated at a video 
rata The system should s^ up a few off*screen planes (image buffers), two for ^odeo 
frame» one for graphical annotation 836, one for text annotation 836, one for Tde- 

10 Pointers 840, one for dynamic marker for dynamic objects 844, and one for dynamic 
marker for steady objects 844. One video frame buffer is a duplicated copy of the other 
for restoring purpose. This will allow the system to offer the ability to erase gnqducal 
annotation separately from text annotation and dynamic maricers, as well as the ability 
to {MTovide timely annotation/Tele-Pointer update without requiring a &st video frame 

15 rata In onier to provide timely annotadon/Tde-Pointer updates, the system has to 
update annotation and Tde-Pointers on a regular basis indqiendently of whether the 
next video frame is ready. During the update of annotations and Tde-Pointers, it is also 
responsible for restoring part of the badcground video frame that is uncovered by the 
motkm of Tde^imers and the erase of graphical/text annotations. This is the reascm 

20 for having two vkieo frame buffers in accordance with the prindples of the invention. 

The video annotator 824 receives local client's and colleagues* acdve 
annotation records and conmiands to draw on the annotation overiay 836. Unlike bcai 
VCR control cmmiands vtdiose action is delayed, the ^stem in accordance with the 
invention handles bcai client*s annotation drawii^ immediately. This is because 

25 ptmiafi feedback is more important As was described eariier, in accordance with the 
principles of die invention, the drawings* text, and Tde-Pointers related to the same 
particqiant will be drawn in the same color distinctive from other participants. 

The a ttach e d tools 832 are components in accordance with the principles of die 
tnventiCMii that can hdp participants to browse video content and to create dynamic 

30 markers fiir static as well as dynamic objects. For example, a vi(feo browser displaymg 
a list of thumbnail pictures representing the video is one such tool. If a participant 
makes any selection on the video browser, this video browser will send a VCR 
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command to the session manager to allow the player to play fix)m this particular pdnt. 

Another example is a tool for creation of dynamic markers for dynamic objects. 
A participant will first select two frames in the video and draw the boundary of this 
objea in these two video frames. The request to find the location of this object in 
5 bttween these frames will be sent to the server. After the server finishes the request, it 
will send a message containing locations of this object in all in-between frames to all 
participants. Each participant will record this information locally and di^lay markers 
whenever the video plays through such frames. Fmally, to create dynamic markers for 
static objects requires a participant to first pause video, elide on the tool button, and 
10 draw a curve. An animation of an arrow following the curve will be shown until a 
participant plays the video again. 

In accordance with the principles of the invention, the Unified 
Computer/Internet Tdephony Interfiu:e 808 provides an unified inter&ce to computer 
or bitemet tdeiriicmy, vrfiich allows users to work with heterogeneous environment 
IS Fig. 9 iUustrates the system message flow in accordance with the prindples of 

die invention, where there are three dients in the collaboradve session. Each dient 
maintains its own message queue (900, 908, 9I6X whm new messages may arrive at 
slightly dififerent time but their order is preserved across all clients. These messages 
indude k)ading of a vkieo or recorded annotation file, modified VCR commands^ 
20 annotation records/commands, Tde-Pdnter records^commands, dynamic marker 
recGrds/bommands. Each session manager (924, 936, 948) is responsible for forwarding 
various messa^ seat by eadi user interfru:e and attached tools (904, 912, 920) to the 
ooiUx»ative dynanw video annotation server (960) and for retrieving messages firom 
the message queue (900, 908, 916). Eadi retrieved message is then filtered throu^ a 
messa^ fiher (928, 940, 952) within the session manager (924, 936, 948) bef(He hoBg 
distrftuted to cfifiefent diem modules for execution (932, 944, 9S6). Duriiig die 
pta^back of a reconted annotation, the collaborative dynamic video annotatkm server 
(960) prinmriiy retrieves niessages from the conference recording (964X but it ^ 
receive mess^ sent by each session manager (924, 936, 948). Each messa^ 
recorded in the conference recording (964) precedes a time stamp. The collaborative 
dynamic video annotatim server (960) retrieves these messages accor(fing to the time 
stanqp. The messages coming fiom both sources are mixed and sent to each session 
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manager (900, 908, 916) in the order the message is received. This ability to mix both 
recorded messages and live messages allow a user to annotate during the playback of a 
recmied annotation. 

Fig 10 illustrates an example showing how event messages are communicated and 
5 a£fea the appearance of each client soeea There are two clients in the conference in tMs 
example The vertrcal direcdon 1020 indicates the time line, whereas the horizontal direction 
lists the cdlaborative dynamic video annotation server (server) and different dient-side 
sy^em components such as the Td&foitm^ Video Annotator, Video Player, User 
Interfioe» and Session A&ns^. Two sets of screen dumps, 1000, and 1010 are di^Isyed 

10 on the left and the ri^ ade of this diagram respecdvdy, in die timeline order. Diffimt 
types of typical messages 1030, such as Join, Play, Show Tele-Pointer, Annotate, Eraser 
Pause; are illustrated in the middle section of this figure, again in the timeline order For 
siniplidty reasons; only abbreviated message nanies are used in this ill 

The present invaition provides a new apparatus for permitting different people 

15 fiiom di£farent places to share their views about a video content, to record such a 
collaborative session^ to ipisy back synchronously such a recorded annotation, and to 
annotate^eccMrd during playbadc of a recorded annotation. The apparatus in accordance 
whh the invention provides an environment for ^chronous collaboration of any video 
content over heterogeneous n^works. It is however possible to use the same apparatus 

20 to annot a t e a video or to piaybadc a recorded annotation in a stand-alone scenario as 
well. 

It is also understood that the apparatus may utilize a televidon receiver 
apparatus for the video displ^ fonction, in association with a computer and a simulated 
or actual hardware VCR or vkleo piayer6recorder. 

25 As was stated above, the invention features a shared "video player*' with nnilti- 

point VOt oontroi, ^namic speed adjustments and the ability to show dynamic 
markers. Any user can play or stop the video, jump to a different locatk>n in the video, 
or djOKunically cfaai^ the play speed, at any given time. Different (rfayers are 
synchronized at the same video frame whenever any VCR activity occurs. Dynamic 

30 martos are automatically drawn when the player displays those frames maricers are 
attached to. Second, die video player window acts as a shared v^Aiiteboard with Tele- 
Pointer support, even if there is no hardware graphic overlay support This shared 
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wlutd)oanl supports free-hand drawing and text. Any user can erase graphical or text 
annotation at any time. Third, the apparatus provides an easy way to attach any toob to 
the shared video player. When such tools are invoked, they will be activated on the 
side; while a collaborative session may still continue. All computation intensive 
5 are done on the server side, without afifecdng the performance of the client. Fourth, all 
conference activities including joint voice comments can be recorded for later 
playback. These activities are time stamped to support synchronized playback. Rfifa, 
any recorded conferoice session can be loaded and played synchronously on screens of 
all users with multi-point VCR control. The recorded joint voice comments are nuxed 

10 with the audio track of the video during the playback. Sixth, any user can still armotate 
duriiig the playback of a recorded annotation in a collaborative session with separate 
VCR controls. Finally, the new annotation together with the playback of a recorded 
aimotation can again be recorded. 

The invention is intended for implementation by computerized iqpparatus, 

15 preferably a p r ogra mm able d^ptal computier, as is per se wdl known in the art, with 
app n qyiate software; in ccnyunction with appropriate peripteral ecpiipment as 
hereinabove described, for performing steps herein disclosed for practidrig the present 
invention. 

As will be understood, the mventicm has been described by way of non-limitiflg 
20 exmsfiBiy embodiments. Various modifications and additions will be apparent to 
diose skilled in the art to wMch it pertains. For example, if hardware implmentation is 
utilized to support vkleo overlay and audio mixing, then tte same fonctions nuqr 
no 

lor^ be required in the form herein described by way of exemplary embocttments; 
25 such asTde-Pcwiter, Video Annotator, audk> mbcer, and graphics overiay image mbcer 

Such changes and modifications are omtemplated to be within the spirit of Ae 
inventim and the scope of the daims which follow. 
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CLAIMS 

What is claimed is: 

5 1 . A method for dynamic video annotation among a plurality of users at respective 
locations, utilizing programmable computer apparatus with information storage ami 
retrieval capability, said method compri^ng the steps of: 

selecting a n^woric service coupled to said computer, 
peifbrming one of (a) starting and (b) joining a collaborative session arnoi^ 
10 said users; 

loading one of (a) a video and (b) a recorded annotation file; 

petfinrming annotation of at least one of graphical, text and audio annotation; 

and 

storing said collaborative session. 

15 

2. A niediodfiir dynamic video annotation as recited in claim Uinchidingastep 
of playing back said collaborative session in a synchronous mode for ones of said users. 

3. A method finr dynamic video annotation as recited in claim 2, y/hemn said 
20 syncfarmious mode is loosely synchronized. 

4. A method for dynamic video annotation as recited in claim 1, including a step 
of storing or reoordti^ visible activities and any voice comments occurring durii^ said 
coDaboiBtive session. 

25 

5. A niethod fir (fynaniic video annotation as recited in claim U whereinsatd 
neiworic seime comprises at least one of Internet TCP/IP, IPX, Modem, and Serial 
comection. 

30 6. A method fordynamic video annotation as recited in claim 1, wherein said step 
ci peribrming annotation comprises utilizing at least one of Tele-Pointers; video 
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recorder/player controls; video browsers; graphical text, and audio annotation; and 
dynamic nnarker creation tools. 

7. A method for dynamic video annotation as recited in claim 6, wherein said 
5 video recorder/player exhibits play and pause modes and said step of perfonming 

annotation can be performed independently of whether said video recorder/player is in 
dther of said modes. 

8. A method for dynamic video annotation among a plurality of users^ utilizing 
1 0 progranrnmble computer apparatus with information storage and retrieval capabilhy, 

said method conq)rising the steps of: 

selecting a n^work service coupled to said computer, 

joining a collaborative session among said users; 

loading one of (a) a video and (b) a recorded annotation file; 
1 5 performing annotation of at least one of graphical, text, and audio annotation; 

and 

storii^ said collaborative session. 

9. A method for dynamic video annotation in accordance with claim 8, inchiding 
20 die steps q£ reoorcfing and (riayii^ back vuieo from a video reoordei/player coupled to 

said computer appaiatus and having a user-operable control interfitoe; 

10. A method for dynamic video annotation in accordance with claim 8» including a 
step of individuals of said users annotating said video as to grqihics; audio; 

25 TeldPcmiter, and text 

11. A method for dynamic video annotation in accordance with claim 10,including 
a step of individuals of said users annotating graphics on an overlay. 

30 

12. A method &t (fynamic video annotation in accordance with claim 10, inchiding 
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thestepsof 

individuals of said users using a Tele-Pointer, and 

automatically restoring video and graphic portions covered over by said Td&- 

Pointer. 

5 

13. A method for dynamic video annotation in accordance with claim 9, including 
thestepsof 

providing a time-stamp all messages reaching said computer apparatus from 
said plurality of users; and 
10 sendii^ ones of said messages having been annotated and provided with said 

time-stamp to said users during said playing back. 

14. Apparatus for dynamic video annotation among a plurality of users; said 
appa ratu s comprising: 

1 5 programmable computer q)paratus with information storage and retrieval 

capability; 

a user interfiice ooufded to said computer apparatus for performing selecttcm of 
a networic service, 

a user interfiK:e coupled to said computer apparatus for performing one of (iO 
20 starting and (b)joiiiing a coBaborattve session among said users; 

a user interfiice coupled to said computer apparatus for loadmg one of (a) a 
video and (b) a recorded annotation file; 

a user inter&ce coupled to said computer apparatus for performing annotation 
of at least one of graphical, text, and audio annotation; and 
2S a user inter&ce coupled to said computer apparatus for storing said 

collaborative sessbiL 

15. .apparatus for c^noamic video annotadon among a pluraUty of users in 
accordance widi daim 14, including a shared video recordei/player coupled to said 

30 OMiqputer apparatus and havh% a user-operable control interfiice, for recording visible 
activides and any voice cmunents during said collaborative session and playii^ back 
recorded annotaticms. 
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16. Apparatus for dynamic video annotation among a plurality of users in 
accordance with claim 14, including apparatus for enabling said users to annotate 
graphics, audb and text. 

5 

17. .^yparatus for dynamic video annotation among a plurality of users in 
accordance with claim 16, including apparatus for enabling said users to amiotate 
gnq^hics on an overlay. 

10 18. .^;>paratus for dynamic video annotation among a plurality of users in 

accordance with claim 17, including apparatus for supporting use of a Tele-Pointer and 
for restoring video and graphic portions covered over by said Tde-Pointer. 

19. Apparatus for dynamic video annotation among a plurality of users in 

15 accordance with claim IS, including i^paratus for enabling said users to annotate ones 
of sud videos. 

20. Apparatus for dynamic video annotation among a plurality of users in 
accordance widi daim 19, including apparatus for providing a time-stamp all messages 

20 reaching said conqputer apparatus from said plurality of users and sending said ones of 
said messages having been annotated and provided with a time-stamp to said users 
duiii^ said playing back so as to permit individual ones of said plurality of users to 
annotate during said playing bade of recorded annotations. 

25 21. A|qi»ratusforenaUingaphirality of users at respective locations to part^^ 
in a collaborative session reganUr^ content of a video; to record such a oolldiorative 
session; to annotateAeoord during playback of a recorded session; and to bade 
synchronously such a recorded annotated session, said apparatus comprising: 

a shared video pl^et/recorder fonction (VCR) available to each of said users» 
30 with multi-point VCR oontroi exhibiting dynamic speed adjustment, and an ^tty to 
show dynamic markers; 

a fimction by which any of said users can play or stop said video; jump to a 
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diffmnt location in the video; dynamically change video play speed; 

said shared video player/recorder fiinction available to each of said users being 
^ndvonized at the same video frame whenever any VCR activity occurs; and 

apparatus for di^laying a dynamic marker when a frame to which such a 
5 maricer is attadied is displayed. 

22. Apparatus for aiabling a plurality of users at respective locations to participate 
in a collaborative session r^anling content of a video; to record such a collaborative 
session; to annotateAnecord during playback of a recorded session; and to play back 

1 0 synchronously such a recorded annotated session, said apparatus comprising: 

a shared video ph^/recorder fonction (VCR) exhibiting a window available to 
eadi of said users, with muki-point VCR control exhibiting dynanuc speed aHj»stniffnt, 
and an ability to show dynamic markers; 

a fiinction by which any of said users can play or stop said video; jump to a 
1 5 different location in the video; , dynamically change video play speed; 

sakl diared video playerAecorder fiinction available to each of said users being 
synchronized at the same video fiame whenever any VCR activity occurs; 

apparatus for displaying a dynamic marker when a fiame to which such a 
marker is attached is displayed; and 
20 said shared video playei/recorder fiinction window acting as a shared 

whiteboanl with Tde4*dmersiq>portfi9r supporting fim-hand drawing and text 

23. Afqmratus forenabliiig a phiraiity of users at respective locations to participate 
in a collaborative sesskm regarding content of a video in accordance with claim 22 

25 lA^ierein said shared vkteo phqreiAecorder fiinction window acting as a shared 
whitdioard with Td&-Pdnter support for supporting fiiee-hand drawii^ and text so 
fiinctions. independent^ of the iMresence of any hardware gra{rfuc overl^^ 

24. Apparatus fiir enabling a phiraiity of users at respective locations to partidpate 
30 in a collaborative sessbn regardii^ content of a video in accordance with claim 22 

wherein any user can erase graphical or text annotation at any time. 
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25. Apparatus for enabling a plurality of users at respective locations to participate 
in a collaborative session r^ardii^ content of a video in accordance with claim 22 
induding means for attaching any of a plurality of tools to said shared video pls^ 
fimcdon. 

5 

26. ^>paratus for enabling a plurality of users at respective locations to partidpate 
in a collaborative session r^arding content of a video in accordance with daim 22 
wherein when any of such tools are invoked, they will be activated on the si^ 
collaborative session is permitted to continue. 

10 

27. Apparatus for enabling a plurality of users at respective locations to partidpate 
in a collaborative session n^arding content of a video in accordance with daim 22 
wherein said collaborative session activities including joint voice comments can be 
recorded for later playback. 

15 

28. AiqpeiEtus for enabling a phirality of users at respective locations to partidpate 
in a collaborative session reganling content of a video in accordance with daim 27 
wfaeiein said activities are time stamped to support synchronized playback. 

20 29. Apparatus for enabling a phirality of users at respective locations to partidpate 
in a collaborative session r^ardtng content of a video in accordance with daim 28 
wtmrein» sudi a reomied session can be loaded and played synchronously on scmm of 
all users with nnuhi-point VCR control. 

25 30. Apparatus forenabliiig a plurality ofusers at respective locaticms to partidpate 
in a collaborative sessbn regardmg content of a video in accordance with daim 29 
wherein recorded joint voice conmtems are mixed with the audb track of a video 
durii^ playback. 

30 31. Appanitus for enabling a plurality ofusers at respective locations to partidpate 
in a ooUaborative session regarding content of a video in accordance with daim 30 
wherein any user can still aimotate during the playback of a recorded annotation in a 
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collaborative session with separate VCR controls to fonn a new annotation. 

32. Apparatus for enabling a plurality of users at req>ective locations to paitidpate 
in a collaborative session regarding content of a video in accordance widi claim 32 
5 wherein said new annotation together with playback of a recorded annotation can f^s^^ 
be recorded. 
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